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ABSTRACT 

This is one of a series of reports based on an 
ongoing reality test of systemic evaluation for instructional 
decision making. This feasibility study is being carried out by the 
Center for the Study of Evaluation with the Laboratory in School and 
Community Relations at a suburban Los Angeles high school (called 
Site A). Viewing a school as a cultural/ecological sys.tem, systemic 
evaluation is a set of principles to guide local development of 
processes to gather, organize, and utilize information relevant to 
the needs and values of the system and its members. This paper 
examines methodological issues derived from a multilevel analytic 
perspective on local school improvement. A multilevel analytic 
methodology examines data from the different levels (student, 
classroom/teacher, school, district) of the educational system and 
their interconnections. It employs test analysis procedures that 
reflect instructionally sensitive performance variation. It links the 
conceptualization of an educational process to its measurement and 
analysis at various levels. It uses analytical procedures that 
potentially identify effects at and within each level of the 
educational system. Numerous tables and a three-page list of 
references conclude the paper. 
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School systems have had rich information environments for quite 
some time. One need only browse the list of information typically 
available in local school districts (see Table 1) to realize how much 
data is routinely gathered and maintained in some fashion. Until 
recently, however, many school districts lacked the financial and 
human resources (and often, the incentives) necessary to nake the 
available information serve as a viable component of decision-making 
In ongoing, enduring school improvement efforts. 

Obviously, times have changed as is evidenced in other papers in 
this symposium and related work (e.g.. Bank and Williams, 1983). The 
collection, analysis and naintainence of both achievement and 
non-achievement data within a comprehensive information system in 
order to (1) examine the functioning and impact of existing school 
programs, (2) monitor key school "health" indicators and (3) plan, 
guide and examine new instructional improvement initiatives is 
becoming a common feature of the local educational scene. These 
activities are both a sign of the times (once again, education and 
educational improvement matter) and an indirect by-product of the 
technology explosion and the improved sophistication of LEA research 
and evaluation personnel that resulted from evaluation and testing 
requirements associated with compensatory education programs. If 
Ijrons, Doscher, McGranahan 5 Williams (1978) were to replicate todj^^ 
their study of evaluation practices in school districts, lack of 
computer resources and expertise and staff technical skills would be 
much less severe problems. To a great extent the knowhow and 
wherewithal exist within school districts to make their available 
information useful for decision-making with respect to a variety of 



Mow that the expertise and technology are possible, it seems 
appropriate to devote more attention to fine-tuning local school data 
analysis and reporting to make practices in these areas (a) more 
sensitive to the substantive decisions school personnel must make and 
(b) better reflect the methodological state of the art. To this end, 
this paper examines selected analytical issues that arise in making 
information obtained from the multiple leveis (pupil /parent, 
teacher/class, school, district, community) of local school settings 
useful for decisions at the various levels* 

Even after a local school community has decided to undertake 
school improvement and allocates the resources to develop information 
systems for use in their efforts, questions remain about how the 
relevant information should be treated analytical ly* Although roost 
districts routinely collect much of the pertinent data for school 
improvement efforts, these data are seldom analyzed and reported in a 
manner consistent with extant knowledge about the possibilities and 
limits of information from multilevel social structures* Since the 
same data (e*g*, from standardized achievement or competency tests; 
parent, student, teacher, and administrator surveys; archival and 
demographic records) can take on different meanings when analyzed and 
reported at different levels (e.g., class vs. school ) and by 
different indicators at the same level (e.g., the school average vs* 
the proportion exceeding a specified level of mastery), comprehensive 
information systems need to be designed to make valio^, pertinent 
knowledge accessible for constituents at the multiple levels. This 
concern for knowledge utility and accessibility may require that the 
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same data be reconfigured (or simply reported differently) for 
different users. Moreover, multilevel analyses, albeit handled in a 
technically appropriate manner, must be reported In a form suitable 
for the analytically unsophisticated. 

In the remainder of the paper, we briefly discuss several 
methodological issues derived from a multilevel analytic perspective 
on local school improvement. This discussion draws heavily from both 
rny previous work on analysis of multilevel data (Burstein, 1980, 1981 
in press; Burstein and Linn, 1982; Burstein, Linn X Capell, 1978; 
Burstein, Miller, & Linn, 1982) and from a longer conceptual synthesis 
that attempted to apply the general multilevel method framework to the 
local school improvement context (Burstein, 1983)* The examples cited 
are taken from both applied resea^rch and from school district 
information and practices (both hypothetical and real). 

The Nature of a Multilevel Perspective 
Local school districts engaged in school improvement would appear 
to be settings particularly amenable to adopting a multilevel 
perspective regarding the collection^ analysis, interpretation and 
reporting of information on school contexts, programs, and outcomes. 
To be internally consistent with this perspective, it is necessat^y to 
employ an analytical methodology that examines data fi*om the different 
'levels (student, classroom/teacher, school, district) of the 
•educational system and their interconnections* Such a methodology 
incorporates both a multilevel conception and an accompanying 
willingness to disentangle affects from a variety of sources and 



levels (Figures 1 and 2' depict the data sources, data domains, and 
aggregation levels likely to be of interest in most school settings.) 
One miist begin with a belief that no level of the educational system 
is uniquely responsible for the delivery of and response to schooling 
and thus substantive questions should rarely be confined to a single 
level (Burstein, 1980, In press; Rogosa, 1978). Thus a multilevel 
perspective and associated investigation focuses on the interface of 
Individuals and the "groups" to which they belong and on the 
implications of this interface for understanding schooling. 

There have been, a number of syntheses of relevant research that 
focus on the theoretical, conceptual, and empirical bases for the 
Impact of the multilevel character of educational systems on the 
measurement and identification of the antecedents and correlates of 
educational performance (Madaus, Airasian S Kellaghan, 1980; Barr & 
Dreeben, 1983; Bi dwell & Kasarda, 1980a, 1980b; Burstein, 1980a. 
1980b; 1983, in press; Cooley, Bond & Mao, 1981; Cronbach, 1976; 
Miller, 1981). These authors build a case that schooling can be 
better understood by, among other things, 

* utilizing an array of group-level (class, school, etc.) 
indicators that are potentially sensitive to differential 
performance associated with differential resource allocation 
strategies • < 

" employing test analysts procedures that are likely to reflect 
instructional ly sensitive variation in performance 

* linking a conceptualization of an educational process to its 
measurement and analysis at various levels 

using analytical procedures that potentially identify effects 
at and within each level of the educational system 



Each of these points Identifies analytical issues addressable 
from a multilevel perspective that becomes especially salient in 
information-rich school improvement contexts. In the remainder of the 
paper, we discuss and illustrate each, point as it might arise as par.t 
cf a school improvement effort. In doing so, we focus on group/ 
social /organizational rather than individual /clinical uses of 
information for class-level, school -level and ssuUI -school 
decision-making^. 

Alternative Indicators of Group Performance 
Much of the analysis and reporting of achievement .and 
non-achievement (e.g., attendance, taking of advanced course-work) 
indicators of educational performance is conducted at the group 
level (typically school or classroom). All too often these 
group-level analyses employ only measures of central tendency (such as 
means and medians^ average percentages). But when one's purpose is to 
understand schooling and depict its consequences, such measures of 
central tendency can hide important differences in the distribution of 
pupil performance and educational experiences. Under laany 
circumstances, the distribution of performance from an tRstructional 
setting is likely to be as informative about the operation of 
educational processes as the group's typical performance (Brown & 
Saks, 1975; Burstein, 1980; Burstein & Linn, 1982; Cooley & Lohnes, 
1976; Klitgaard, 1975; Linn & Burstein, 1977; Lohnes, 1972; Spencer, 
1983; Hi ley, 1970). 

The point here is that although a concern for achievement may 
drive instructional improvement efforts. It is important to keep in 



the local school context and be more refined in its objectives. Under 
usual schooling conditions, a focus on raising the performance level 
of students around the middle of the overall performance distribution 
(e.g., say 40th - 60th percentile) will yield the highest gains in 
mean performanceZ. Yet such a thrust ignores just the segment of 
students who have the greatest needs. A focus on the performance of 
the lowest quartile, on the other hand, devotes instructional 
resources in a manner likely to reduce the spread of performance (by 
establishing a performance floor or boosting more students over the 
minimal mastery point). Thus,. the multilevel principle that group 
means do not account for all relevant group-level information should 
lead to context-sensitive analyses intended to monitor instructional 
improvement^. 

Examining Spread 

The hypothetical data in Table 2 illustrates the value of 
multiple group-level indicators in simple comparisons of class-level 
performance. While the students in both classes started at the came 
'level on the pretest and had the same mean performance on the 
posttest, the variation in posttest performance is much larger for 
Class 2. Instruction in this class led to differential increments in 
learning gains (some students learned more than their counterparts at 
the same pretest performance in Class 1 while others learned less) 
while the posttest performance patterns in Class 1 reflect a more 
uniform distribution of learning gains. One. needs to know more about 
the specific circumstances to judge whether one class-level 
performance profile is to be preferred over the other, but the two 
profiles are definitely different and should be treated accordingly, 
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Comparing Distributions 
^ The data reported in Table 3 (taken from Spencer, 1983) 

illustrate the Importance of trying to capture the entire distribution 
of performance in comparing groups (schools in this case)^» ^.f 
Judgments about a school's effectiveness were based on a criterion 
score of 70, School C would be top ranked (because 20 percent of its 
students exceeded this score) while School A would be ranked at the 
top ii the cutoff were either 45 or 40^ This example might seem 
far-fetched, but it depicts what can happen when schools are judged by 
either the number of students taking advanced placement courses or 
their average SAT score versus judging schools according to the 
proportion of students passing a staters minimum conpet»:ncy or high 
school proficiency test. 

Measuring School Effectiveness by Subgroup Comparisons 

Another actual example derives from the recent school 
effectiveness literature* In his studies, Edmonds (1982) focussed on 
wi thin-school differences between lower SES and higher SES students in 
the proportion of students achieving mastei^ of designated educational 
objectives. Thus, if a substantial proportion of a designated group 
within a school performed at the prescribed level or if the 
differences in proportions achieving mastery were approximately equal, 
the school was judged "effective*. In essence, Edmonds* interest in 
the antecedents of effective achievement of low-income students caused 
him to depart from typical practice in school effectiveness studies of 
concentrating on school mean levels across all pupils. 
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Characterizing Survey Responses 

The last example is taken from the reporting of responses to 
survey items collected as part of A Study of Schooling (Goodlad. 
Sirotnik, & Overman, 1979) and later provided as part of a feedback 
package for a school (Table 4). Considering the student responses in 
Exhibit A first, the mean response to every item provides little 
Indication of the diversity of feeling expressed by students, A 
significant number of students strongly dislike each activity, while 
In the case of "working with the whole class", an equal nuniber 
strongly like this mode of learning. Students indicated definite 
preferences that, if reflecting true feelings, dictate against 
concluding that either it doesn't matter or students are undecided^ 

A somewhat different point is illustrated by the teacher 
responses to the organizational problem solving and principal 
leadership dimensions. The dimensions yielded equal means, but 
teacher opinions are certainly more divided with regard to their 
beliefs about principal leadership. The staff cohesiveness dimension 
exhibits both a higher mean and the virtual absence of "disagreement" 
responses (only Z% scoring either 1 or 2). The means alone simply 
cannot capture what these data tell us about teacher perceptions. 

There are other studies and other group-level Indicators (e.g., 
Burstein & Linn, 1981; Burstein, Linn, & Capell, 1978; Burstein, 
Miller, 4 Linn, 1981) that offer possibilities for improving 
explanations of the relationships among educational contexts, 
processes and outcomes. How to adapt at least the logic, and perhaps 
the metl-ods, of alternatiye group-level indicators for use in local 
school improvement activities clearly warrants further consideration. 
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Instructlonally Sensitive Test Analysis 
Clearly, the emphasis In test construction, analysis, and 
Interpretation is on individual differences in both classic and IRT 
psychometric treatments of test data. Yet it would seem that a 
concern for the sensitivity of test performance to instructional 
experiences would require test analyses approaches that reflect the 
organization of instruction and the circumstances in which students 
receive instruction. There is ample evidence of the substantial 
variability across classrooms and schools In their content selection, 
emphasis, coverage, and method of coverage, not to Mention the quality 
of instruction. Under these circumstances, multilevel examinations of 
the patterns of test performance (at the class and school level in 
addition to the student level) can be potentially valuable for 
detecting effects to background differences (e.g., prior learning, 
socieconomic and demographic differences), instructional coverage and 
emphasis, and Instructional organization {e.g., grouping and pacing 
effects). When these separate effects can be identified, it my then 
be possible to construct measures and indices whicu are sensitive to 
the context factors of instruction and describe performance 
accordingly. 

We illustrate multilevel appraches to understanding and 
describing test performance in two ways. First we present a 
hypothetical example of the hazards of basing interpretations of group 
(school, class) test performance on horizontal aggregates of many 
subskills and competencies. This example is followed by a brief 
report from an investiaation of class-levpl 
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test items that, in our view, illustrates what can be gained from a 
multilevel analysis of patterns of responses to test items. 

Aggregating Over Test Content 

The proper handling and interpretation of scores aggregated over 
individuals has received most of the attention in jmilttlevel methods 
literature. Yet, in several respects*, the logic holds as well for the 
content of instruction and of outcome measures. Tiie choic^i between an 
emphasis on basic skills or on a broader arr^ of knowledge has much 
In common with the decision about which group or level is of 
interest. Just as a focus on low income students at the school level 
dictates interest in certain indicators of performance (and perhaps 
disinterest in others), judgments of the success of school improvemejit 
efforts can depend on the chosen level of aggregation over the content 
of instruction. It can also depend on the fonn of measurement of the 
content. 

The concept of level of aggregation and measurement of 
instructional content can be depicted as follows. Following a literal 
interpretation of the dictums of Title I, School A institutes a strong 
back-to-basics effort, adding more drill and practice activities 
(spelling quizzes, timed math drills, memorization and recitation of 
poems). The additional time School A devotes to these activities is 
• obtained by foregoing most social studies, arts, science, and music 
instruction. School B, on the other hand, increases enrichment 
activities and attempts to enhance the breadth of its curriculum- 
through dramatic play in its social studies work (e.g., 
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various classes enact political campaigns prior to major elections, 
operate mock city governments, "live" through the experience of the 
Pilgrims, etc.). The teachers in School B tie in roost lessons in 
reading, mathematics, and writing with these dramatic assignments but 
leave little time during school hours for drills on math, spelling and 
language arts fficts. 

Schools A and B make clear th^ir instructional preferences. 
However, unless the array of non-teacher made tests (annual 
standardized tests, state u-sessment, -Sistrict continuum) given in the 
schools are unusual, aggregate scores from these tests will 
differentially reflect the instructional emphasis and quality of the 
two schools. If two-thirds of a test's reading questions were devoted 
to word identification, vocabulary and spelling and its math questions 
to basic facts and computations, total test scores might make it 
appear that School A is more effective because tts students uniformly 
mastered their narrower curriculum material that predominated in total 
scores, while School B's students performed more inconsistently on 
their facts and mechanics. (Presumably, School B would do much better 
on more complex comprehension, reasoning and interpretation material 
that is covered by a more limited portion of the test.) 

Unfortunately, this fictional scenario probably occurs all too 
often in current efforts to determine the content of instruction and 
its measurement in school improvement efforts. The premise here is 
that the level of refinement in distinctions about instructional 
content should be an explicit choice and the measurement of the 
consequences of instruction should be sufficiently refined to reflect 
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both desired and unintended content distinctions. Aggregation over 
content in the scores from multidimensional tests ts potentially 
hazardous if the intent is to determine what has been taught and how 
wel 1 . 

Examining Response Patterns at the Group Level 

Examinations of the patterns of students' responses to test itms 
across occasions and across groups (classes, schools, etc«) can be a 
particularly informative means of deriving explanations of test 
performance. A reanalysis of selected test item data from the 
Beginning Teacher Evaluation Study (BETS; Fisher et aU> 1978) 
suggests how response patterns might provide information about 
instructional differences. Miller (1981, 1984-) examined answers 
chosen by 123 students from 21 fifth-grade classrooms to the 15 items 
from the fractions subtest on two occasions (prior to (Occasion B) and 
following (Occasion C) most instruction in this subject area). He 
classified the test items into four subtopics: adding of fractions, 
subtracting of fractions, equating fractions, and solving fractions 
with algebraic unknowns (e.g., X/3 = 6/9. What is X?). Tables 5 
through 7 present a subset of Miller's results*. 

First, the intercorrelations of item performances between classes 
and among students within the same class (Table 5) clearly indicate 
the effects of differential topic coverage across classes and within 
classes. Virtually all of the high item intercorrelations occur in 
the betv/eeh-class analysis. This reflects differential coverage of 
topics across classes*. Those classes which taught addition of 
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fractions also taught subtraction ^nd those classes wtth high (low) 
performance on addition also tended to exhibit high (low) performance 
on subtraction. The classes which taught algebraic unknowns 
apparently didn't teach (or unsuccessfully taught) addition and 
subtraction of fractions. On the other hand, there was virtually no 
intercorrelation among item performance within classes. There was 
very little tendency for a student to perform well across all items/ 
Different students answered the various questions correctly 5. 

Table 6 present results from a selected set of classes with 
distinctive response patterns. Since time allocated to fractions 
instruction per week is also reported, it is possible to also 
speculate as to whether classes received any fractions instruction at 
all. Note especially the contrasts in performance. For example, 
•classes 8 and 5 had the same posttest score and approximately the same 
gain, but class 18 did so by covering and mastering every topic except 
addition of fractions (these students exhibited the same performance 
pattern at pretest so it is likely that the teacher simply did not 
cheek to determine whether students had mastered the mechanics of 
fraction addition after the pretest.) Class 5 likely depicts the more 
typical pattern of mastery of simple addition and subtraction of 
fractions and virtually no other content coverage. 

Classes 3 and 16 represent another interesting contrast. These 
two lowest scoring classes got there in different ways. The teacher 
in Class 16 devoted a substantial amount of time to the coverage of 
fractions but virtually all the time seems to have been spent learning 
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about algebraic unknowns. Class 3 devoted almost no time to the 
fractions topic (the majority of the class did not attempt to answer 
the questions at either pretest or posttest.) But there was some 
differentiated content coverage as a few students in Class 3 mastered 
all four subtopics. 

Table 7 presents information derived from examining the 
class-level variation in the actual response alternative selected. 
This analysis clearly suggested systematic differences across classes 
in the types of errors students made. The. two addition-of -fraction 
items demonstrate that students in several classes apparently never 
learned not to simply add numerators and denominators. Several 
classes did not appear to teach the expansion of fractions or taught 
fraction reduction in such a way that the students did not grasp its 
flip-side. The algebraic unknowns items exhibit the greatest variety 
of class-specific confusion. Obviously, some aspect of instruction ts 
responsible for the systematic misunderstanding of how to do this type 
of problem. It may simply mean that in the absence of Instruction on 
algebraic unknowns, students from a given class facing a novel task 
(for them) try to apply some other algorithm they have learned. 

This example of investigating patterns of test performance barely 
scratches the surface of modern psychometric work on item response 
patterns (See, e.g., Harnisch, 1983; Harnisch & Linn, 1981; Sato, 
1975, 1980; Sato S Kuto, 1979; Tatsuoka & Linn, 1983), much less 
recent advances in information processing models for test design and 
interpretation (e.g.. Baker & Herman, 1983; Brown S Burton, 1978; 
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Btrenbau™ i Tatsuoka, 1982; Curtis S Slaser, 1983; Da,fs 1979- 
■Tatsuoka. 1983). Yet ,t ,s clear that the co*,„at,o„ of thts'^odern 
onentatfon toward test design, better psychcetrfc Indices of 
response patterns, and analyses that take Inte const deration the 
levels at which Instruction Is delivered can provlde better 
information about test performance for declslonn^akers. 

Mtllevel Measure m ent of Education.! p^. rnnl, .1 

The principle that the same observable variable can measure 
different constructs at different levels of analysis Is well- 
established (Bursteln. 1980a. 1980b; Bursteln. Fischer, and Hlller 
1980; Capell. 1981; Cronbacb. 1976; sirotnlk. 1979). A few examples 
serve to emphasize Its ubiquity in educational research. Take, for 
instance, the standard measures of socioeconomic background typically 
found in studying schools. At the individual level, they m,y properly 
convey the parental inves^ent in the individual child's learning. 
Once aggregated to the school level, social background measures also 
renect the community context (e.g.. wealth, urbanism, co^nitment to 
quality education) which In many cases conditions the resource 
allocations to schools. Within an educational level, relative social 
background positions students within a potential status hierarchy 
(e.9.. a big fish in a small pond) That can affect their experiences 
(Bursteln. 1980a. 1980b; Bursteln et al., 1980). All three measures 
of social background my be important in understanding the experiences 
and performances of students but they do represent distinctly 
different mechanisms. 
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In a reanalysis of data from an observational study of the 
factors influencing student learning, Burstein (19cO) demonstrated how 
the interpretation of a measure of the relative anxjunts of student 
learning tasks judged easy changed as the analysis shifted from the 
student to the class level. Students' success rates in learning tasks 
at the individual level captured proximal student ability and thus 
werepositively related to student performance. At the class level, 
this same observational variable reflected teachers* policies with 
regard to task difficulty and in many instances exhibited negative 
relationships with student outcomes. 

The problems of change in variable meaning across levels are 
particularly evident in the literature on organizational and 
educational climate (e.g., Capell, 1979; Sirotnik:, 1979). The 
distinction between a specific student's perception of classroom 
climate, which reflects both absolute and comparative aspects of 
individual personality and perception, and the average perception of 
the class, a normative measure of the instructional environment, is an 
important one. Whether the "organizational" or the "psychological" 
aspect of the climate is most salient in a given context is unclear. 
Capell (1981), for instance, construed aggregate responses of teachers 
within schools on scales purported to measure .the d-egree of innovation 
and teacher influence as indicators of the atmosphere and 
organizational structure of the school program (See Table 8). In 
contrast, the individual teacher responses, relative to the responses 
of other teachers in the school, were interpreted as indicators of the 
teachers' sense of personal efficacy. That the effects of aggregated 
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and individual measures on pupil outcomes were opposite In sign and 
consonant with expectations reinforces the need for a better 
understanding of how aggregation affects the measurement of program 
and process characteristics. 

The studies cited above are important for our present purposes in 
two respects. First, they demonstrate the value of multilevel methods 
in educational research and evaluation. Second, and more importantly, 
the measures used in these studies ~ socioeconomic background 
indicators, survey responses from students, teachers and parents, 
classroom observation data — represent typical information about 
educational processes and contexts that are or can be gathered in 
local educational settings. Apparently, these measures can serve as 
indicators of a variety of constructs — home resources, community 
resources, organizational structure and atmosphere, personal efficacy, 
classroom and school climate, appropriateness of content, student 
level of functioning — that are important in understanding and 
improving schooling when the linkage between the level of aggregation 
of measures and the construct of interest is clear. 

The relevance of these concerns about the shift in variable 
meaning across levels is particularly pertinent to a variety of 
investigations of the effects of schooling that a number of local 
School districts have undertaken. Several school districts have 
conducted their own studies of school effects (e.g., Kean, Summers, 
Raivetz « Farber, 1979 (Philadelphia); Ramey, Hillman & tothews, 1982 
(Seattle)) and school effectiveness (e.g.. White & Kemp, 1976 
(Atlanta); Gastright, 1977 (Cincinnati)). The methodology they employ 
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mirrors the practices of large-scale investigations of school effects 
and school effectiveness. The LEA-based studies seem no more- nor 
less resistant to problems in measuring the variables of interest at 
the appropriate levels, and incorporating them properly in their 
analyses, than their large-scale, multi-site counterparts (see 
Burstein. 1980; Madaus. Airasian & Kellaghan, 1980; and Purkey & 
Smith. 1982 for discussions of methodological problems with these 
types of studies). If local educational agencies use these 
investigations to guide their decisions about instructional 
improvement programs and other school renewal activities, then one 
would hope that inattention to specific concerns about appropriate 
level of measurement and its relevance to construct-indicator match 
would have limited impact. 

# 

Analytical M ethods for Disentangling Multilevel Effects 
Two recent classroom studies of beginning reading demonstrate the 
value of decomposing the variation of students' instructional 
experiences and performance into variation associated with subgroups 
within classrooms. Barr & Dreeben (1983) focussed on content coverage 
as a variable and found that it varied mainly between readina arouos 
and not between teachers. Not surprisingly, students performance also 
varied primarily between reading groups. 

In the Texas First Grade Reading Group Study (Anderson. Evertson 
& Brophy. 1979). the focus was on teaching behavior variables such as 
teachers' selection of students to read (e.g., non-volunteer 
selections), types of student responses to teacher questions, and 
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"^pes of teacher feedback. In a secondary analysis of the study data» 
Martin^ Anderson, and Veldman (1980) decomposed the variation in 
student achievement into effects of teacher behaviors at three levels: 
students within reading groups, reading groups within classes, and 
cl asses » Most of the significant relationships were for 
students-wi thin-reading groups* Also teachers tended to change their 
selection strategies across reading groups within their class. An 
analysis of class means only would have missed the effects of 
teachers' differential activities across reading groups and the 
differential impact of teacher behaviors on the members of specific 
reading groups* 

There is no need to provide additional rationale for and describe 
developments in analytical methods for disentangling multilevel 
effects* As with the studies of reading, the substantive 
investigations described in earlier sections typically combined a 
better conceptualization of the multilevel character of educational 
data with analytical machinery adapted to the substantive questions of 
interest rather than molding the theory to meet the conditions of the 
statistical procedures* Generally, the analytical procedures employed 
were the familiar ones, but these tools were used in a variety of ways 
that better mirrored the process of schooling* Typically, better 
analysis of multilevel educational involve disentangling influences at 
and within each level of the educational system by conducting multiple 
analyses or a common analysis with measures collected from (or 
aggregated to) multiple levels* 
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Currently, methodological research on methods for analyzing 
multilevel data is focussing on relatively sophisticated procedures 
that require stronger assumptions but at the same time, are more 
robust to typical shortcomings in social science data (e.g., missing 
data, asymmetrical distributions, measurement error, 
heteroscedastlcity, resistance to outliers. Relevant work is reported 
i in Aitken, Anderson, and Hinde, 1982; Burstein & Gustafsson, in 

progress; Goldstein, In progress; Msson, Wong & Entwistle, 1983; 
Rachman & Wolfe, 1983; Schneider & Treiber, 1984. Whilelittle of 
this work will have direct bearing on routine analyses to guide local 
school improvement, tliey may provide a better means for 
re-interpreting results from studies of the effects of schooling in 
ways that are more consonant with the perceptions of local school 
personnel. If so, such reanalysis would provide better support for 
effective local practices and less ammunition for school critics. 

Concluding Comments 
The value of a multilevel perspective for understanding the 
effects of schooling is becoming a more commonly held perception 
across a wide array of educational professionals ( researchers , policy 
makers, and practitioners). We have attempted to illustrate several 
ways in which such a perspective might lead to more sensitive and 
sensible data analysis that are better suited to school improvement 
efforts. 

To some degree, our efforts bog down when we attempt to shift 
from studying and understanding the concept of multilevel analysis and 
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turn to Its practice. School cultures involve a variety of sources of 
information and a number of constituencies (e.g., teachers, 
counselors, administrators) with clearly demarcated responsibilities 
that might be able to use properly collected, appropriately analyzed 
and routinely accessible information. But little is known about how 
school building personnel operate in a context with high-quality, 
timely, pertinent information, either because these conditions do not 
exist or no one has yet documented how school -level personnel respond 
under such ideal information conditions. If the other symposium 
Papers are indicative of what is now possible with respect to the use 
of comprehensive information systems in schools, we won't have to wait 
very long to determine whether the promise of a multilevel perspective 
toward the analysis and reporting of school data is real or illusory. 
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FOOTNOTES 

iThis distinction is a useful ona. Individual /clinical uses of 
Information include such activities using test data for Individual 
diagnosis of learning problems, placement decisions, individual 
student counseling and guidance activities, administrator supervision 
of Individual teachers and similar individual personnel matters 
(hiring, course asignments, promotions, etc.). Group/social/ 
organizational uses refers to the myriad of ways In which data from 
individuals are aggregated and organized to characterize/depict/ 
understand the functioning and behavior of groups of individuals. 
Class-level test performance and background profiles, subgroup (e.g., 
by ethnicity, sex, grade-level, curriculum track, SES) information; 
course enrollment and course-taking patterns across subject matters 
and subgroups of students are all examples of the latter. Host of the 
analytical developments pertinent to this paper deal with the latter 
type of use. 

^There are both substantive and technical reasons for this. 

ouw9>.aiibi vcij , 3(.uucii(.d a*, iiigiicr jjcrtcnti le icveiS naVe 3 lOW 

ceiling; that is, much of the ngw material they might learn is not 
likely to be reflected in substantial improvements in the test 
performance because this material typically is not wel 1 -represented on 
the test. Low performance, on the other hand, may require substantial 
resources to boost performance above, say, the 50th percentile and 
thus draw off resources from students in other parts of the 
distribution. For example, concentration on the skills needed by low 
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performers lead to more wait time and inefficierK;y for high 
performers (unless the latter are allowed to "work ahead on their 
own").' Performers in the middle of the distribution are likely to 
benefit from the focus because th^ have sufftcieni fooni to grow and 
may simply require a bit more targeted instruction to clear up certain 
misconceptions (see later example about fraction addition) or to 
acquaint them with topics not previously covered. 

The technical side of the argument is the well-known relationship 
between raw score points and percentiles. It takes more raw score 
change to move up a given number of percentile points in the tails 
than in the middle of a normal distribution. Thus gains in knowledge 
in the middle of the distribution boost the average percentile more 
rapidly than gains in either the upper or lower tails. 

30ne does not have to be devoted to compensating for the inadequate 
performance of low income pupils to derive benefits from interest in 
the distribution of performance rather than simply its level. In 
their annual reports, the California As^ssment Program provides 
schools with the quartile distributions of the performance of 3rd 
grade students along with a variety of mean indicators (overall and 
for varfous demographic subgroups). A former principal of a suburban, 
typically higher performing school pointed out that while the school's 
overall performance each year (typically above 70 per cent correct, 
which is above the 90th percentile statewide) didn't tell him 
anything, he did keep track of the number of children that fell in the 
lower quartile each year because this meant that there were still 
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students who needed to improve. Thus, even high achieving schools can 
benefit from an awareness of the functioning of thetr weakest students 
and school mean performance doesn't typically capture this type of 
information. 

^Spencer's paper (1983) considers a number of statistical problems 
associated with the typical use of test scores for comparisons of 
outcome differences among groups. Ke highlights the problems 
associated with the ordinal properties of most metrics used to measure 
outcomes and presents a case for switching tcj Indices of "stochastic 
ordering" when attempting such comparisons. This shift would 
certainly be in the direction of maintaining more distributional 
information in group-level analyses. 

^The exceptions to the lack of correlation across items within classes 
are typically for those items that are essentially parallel (e.g., 
items 1 and 2; item 6 and 7; and items 11, and 13) or involve 
analogous straightforward topics such as addition and substraction 
items containing a common denominator. 
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Table 1 

The Types of Information Routinely Collected 
(or collectable) in School Districts 

A. Demographic/Archival 

1. Student demographics-age. sex, ethnicity, home language, 
parental occupations and employers, eligibility for AFDC, 
reducted price lunches, medical histories, home address,' 
mobility (how long in particular residence) parental 
educatign, family size 

2.. Teacher and building-level administrator backgrounds - age, 
education, previous employment and educational history, 
special certification and subject-matter expertise 

3. School building characteristics ~ information about physical 
plant (e.g., age, capacity, particular resources), 

4. student body and community composition-ethnic composition, 
neighborhood wealth, community involvement in neighborhood 
schools (e.g., PTA membership) 

B. Financial 

5. Payroll expenditures 

6. Materials and supplies 

7. Equipment 

8. Maintenance 

9. Special programs (e.g. entitlement programs, staff 
developent, remedial services, counseling and guidance) 

10. Transportation 

11. Safety and Security 

C. Testing 

12. Standardized norm-referenced tests 

13. Criterion referenced testing 

14. Minimum competency and proficiency testing 

15. Group and individual ability and aptitude testing — done 
typically to determine pupil eligibility for special programs 
and placement decisions 

16. Teacher-made tests and curriculum embedded tests 
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Program Characteristics and Participation 

17. Special program participation — avallabnity and staffing of 
special programs at local school sites 

18. Curriculum Information — curricular packages and texts used 
in classrooms, topic coverage from continuum (assumed and 
measured) 

19. Course taking patterns ~ information from student cummulatfve 
records and from prescribed offerings 

20. Grading practices — teacher reports of student grades 

Student Performance, Participation, and Behavior 

21. Grades by content area 

22. Participation In extracurricular activities by types 

23. Awards — e.g., scholarships 

24. Absenteeism and tardiness 

25. Reported disruptive and inappropriate behavior 

Af fective, Attudinal,and Observation Information 

26. Student responses to surveys about class and school 
environments and other aspects of their educational experience 

27. Teacher measures of classroom and school climate and 
activities 

28. School building administrator measures of school climate and 
activities 

29. Parental surveys of perceptions and support of school 
activities 

30. Parental participation in school activities (e.g., volunteers, 
fundraising attendance at school functions, scheduled 
conferences) 

31. Administrator observations and evaluations of teachers 

32. Teacher observations of other teachers 

33. District personnel's observation and interviews of building 
personnel 

34. Surveys of graduates to determine occupational and educational 
status 

35. Information about student dropouts 
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G. District Evaluation Reports 

36. Routine annual reports to board and federal and state agencies 

37. Evaluation of specific educational changes 

38. Instances of local school assistance by type and disposition 



^ Source Burstein, L. The Use of Existing Data Bases in Program 
Evaluation and School Improvement (1983) 
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Table 2 



Hypothetical Test Results For Two Classes 
with Equal Pre-and Posttest Means, 
Equal Prestest Variances, but Unequal 
Posttest Variances 



T*.<;f e^nv. Er®*®*J Frequencies Posttest Frequencies 

Test score class 1 Class 2 Class 1 Class 2 



7 ■ 0 0 

6 0 0 

5 2 2 

4 6 6 

3 S 9 



0 3 

2 3 

6 4 

9 5 

6 4 



2 . 6 6 2 3 

12 2 



0 3 



^ 0 0 0 0 

Means X.^ « 3 X.g = 3 Y.^ = 4 Y.^ - 4 

Standard i.os 1.08 • 1.O8 2 17 
Deviations. 



Source: Burstein, L. & Linn R.L. Analysis of Educational Effects from a 
Multilevel Perspective:Disent'angling Between- and Within-Class 
Relationships in Mathematics Performance, CSE Report No. 172, 
Uaiversity of Calif omia Los Angeles, Center for the Study of 
Evaluation, 1982 
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Table 3 > Pere^tit' of Studentai Scoring at Least X 



School 


40 


45 


Scor« X 
30 


55 


V • 

70 


A 


90 


80 


60 


50 


10 


B 

• • 


85 


75 


55 

• 


48 


14 


C 


83 


70 


65 


48 


20 


0 


71 
• 


60 


40 


20 


3 


• 
• 
• 


• 
• 
• 


• 
• 


• 
• 

* 


• 
• 
• 


• 
• 
• 



Source: Spencer B. D. On interpreting test scores as social indicators: 

Statistical consideratidiii. Journal of Educational 
Measurement . 1983, 20(4), 317-334* 
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Table 4 

^nifLnf 5^?!?;"/° ^'i'^^y ^""^ g^!^^^ response Is Insufficient , 

to reflect distribution df responses. 

Exhibit- A-^ Responses, of 26 students from a sJiigle secondary school class . 

Percentage of students responddLng:. . • . *^ ' 
Like Like Dislike Dislike 

very much somewhat • soroevhat . vei^ much Mean 

Working with the . . 

Vhole class 23 38 15 23 2.4' . ^ 

Tell in my aim- words ' . • ' 

what I have learned...... .'04' 23 31 42 h ti 

Do word probleiasf. .1 04 54 X9 23 2 it- 

Do research and write i • • • 

"ports : 04 23 31 42 3.1 

Exhibit B— Responses of 34 teachers from a single school to items measuring 
teachers perceptions of the work environment. The 77 items were 
combined into 3 scales (dimensions of the work environment) which 
were^^labeled "organizational problem, solving"^ "principal leadership" 
and staff co^esiveness". Items were answered on a six-point "agreement 
scale and the school mean and distribution of teacher scores (average 
• response to items from a given dimension) are reported below:. ' 

Dimension ' ' - Mean I ' ^J^acher Distribution (Z) 

r :: . ' r... . ' ' _3 _4 _5- •\j6 - • 

1. Oranizatioual Problem Solving 3.4 3 12 41 29 12 

2. Principal Eeadership " 3.4 12 18 24 26 9 12 

3. Staff Cohesiveness 3,7 0 3 44 35 15 3 

— - — ' — - - - ^ * • * 

a . , • 7Z '• ~ ~ ' ' ' 

Data and questions taken from an example feedback package from A STudy of 
Schooling (Goodlad, Sirotnlj; et. al.) which also appeared as Appendix B in" . 
Sirotnik & Burstein (1983). . .. 
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ITEH 
1. 

. 3 
5 



Item Intercorrelatlons between classes (lower triangle) 
and withijx classes (upper triangle) on occasion G. 



Subtrtctlon 
2 3 1 



.86 
.19 

.43 



..75 

.53 
.41 • 



.07 
.00 

.50 
.36 



.33 
.29 
.30 

.39 



>.03 
•.08 
.06 
.03 



Addition 
7 8 .9 



.51 
.54 
.17 
.25 
..15- 



.29 
.33 
.29 
.30 
'.00 



.09 • .01 
.07 -.00 



.26 
.23 



.25 
'.22 



.37 •*.40 



is. 

• .23 
.16 
.23' 

'.19 

-.or 



Equtttn) 
IL 12 .13 



Algebra te 
HinipuUtfon 



.23 

.19 

.29 

.2S- 

.19 



.21 
.12 
.03 
.05 
.18 



,22 
.02 
.05 
.15 
.19 



]4 

.26 
.29 
..16 
..40 



15 

.20 
.18 
.06 
.26 



_^16 -.01 

.V9" 
.18 
.20 
.24 

'IL 
".23 

.35 

.29 



6 
7 
8 

9: 

10 



11 
12 
13 



14 
15 



.68 .42 

...52 ',29 

* .49 . ■ .29 

'.53 l37 

'.56 .28 



.34 
.55 
*.30 
.'.33 
.05 



.43 
.48 
.48 
.45 
.40 



.27 
■•.38 
.42 
..69 
.49 



.86 
.66 
.64 
.61 



.59 

.64 

.59 

.62, 



.14 .15 
.29 • :25 
.67 

.75 

.53 .69 



.36 
.50 
.24 
.19 



.19 
.19 
.35 
.2) 
.12 



.55 
".41 

.50 



.45 
.3? 
.51 



.33. 

.60 

.19 



.56 
.74 

.45 



.64 
.52 

.60 



.42 
.26 
.26 



.45 
,38 
.16 



.47 
.57 
.59 



.58 
.41 
.51 



.51 
.23 
.29 



.69 
.56 



,03 . .07 
,16 .08 
.29 • ^23 
,,Z\ .16 
.06' - J 3 
.397 
.38 



;S0 
.69 



.05 
.16 

..35 
.23 
.04 
.27" 
.29 

-.17 



.18 
,29 



.07 
.18 



..07 
..05 



.36 
.33 



.31 
.39 



-.10 

;io 



..12 
.03 



.41 
.48 



,35 
.53 



■ .12. 
.33. 



.47 
.64' 



.9U 

.48 



.49 



.81 



.37 



Sourcei M.D. Miller, Item Response and Instructional Coverage, -1984. 
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Table 6 

Cl«s»rooas exhibiting distinctly* cla»3-l«v«X paccccos of pftrfonwinc* on fraction Iteu.' 



Class AUogattd Tlce PogCtasC Cain 

£ — " Hwn Rank Mean Rank^ Mean Rank* N^ttura of tha ittm raspoos^ i^atf m 

18 . U9 1 7.5 10.5 5.0 7 MVstarad aqu^iting, al»abraic manipulation, and siapla 

subtraction. Only probisia at posttast vas addition 
'* of fractions (adding miMrator and dammlnator) vhara 

only 2 corracc out of 30 rasponsas (6 studants raspondins 
to-5 ita«a}« ' * 



5 


77 


5 

• 


7.5 


10.5 


5.2 


5 


Studants can do sivpla addltdon and subtraction Itatts 
but not Mora «o«pUcatad on as. Ko vork on aqua ting* 
lona covaraga of algabralc MnipuUtlon but >asicaUy 
don^t undaratand tha coocapt. 




•114 


2 


12.5 


1 


6.5 


1 


yt.vuA.wMm at pvaccasb ax^apc ott aagaorsxc iaanipuxation» 


12 


87 
• 


4 


11.3 


3 


6.3 


2 


Most aiaatarad avai^thlng but 1 or 2 didn*t mastar 
any fraction topic (possibla tfiffaraWclal covaraga). 


- 11 


•'27 


' 6 


12.3 


2 


1.7" 


15 


lota of tima spant on fractions vlth not such nav 
learning » at loas't on tha topics saasurad by tha taat« 


8 ' 




14 


. 9.0 


7 


5.6 


3 


Succaas on topica covarad (sub traction, aqua ting » si&pla 
addition). Addition othar than siopla conoon danosinator 
ailslearnad or not taught. 


26 


46 • 


10 


6.4 


13 


5.0 


7 


Traction curriculuM at low Icval. Most learned only ^ 
.addition and subtraction vith'coiaBoa danoalnatos. Ona 
or two acudanta laamad mora** ^ 


•14 


29 


12 • 


5.3 


15 


3.7 


9 • 


Piffarantlatad taaching awi laaming. So«q maatarad 
avatychlng, som maatarad addition j|nd aubtraction 
with coapon donomlutor, otbara maatarad' (probably 
covarad) nothing. 


16 


52 


7 


2.8 


. 21 


1.5] 


16.5 


.Studenca in .this daaa only maatarad algebraic manipulation 
they did not anavar any othar quaatlona on althar prataat 
or poattaat (with ona or two exceptions). 


3 


3 


•19 
• • 


4.8 


20' 


3.3" 


10.5 


Dlffarantlotad content coverage ^almoat »11 on algebraic 
manipulation J. aoaa students mastered moat topics. 



— j.^ jtwers iron jli ciasarooffls m tha Beginning Teacher Evaluation STudy.Tbasa 

analyses ara reported in Miller (1984). ^jLu*ti.utt ^xu^j. »«>« 

minutes per week 3pent on fractions during period between protest and posttest. ' ^ ' 
^ed on r^, ot «an tl« .Uoc.ced to frac^on. .cro„ the il cI.„roo«.. \ • 

■^Sa-ed cn po,tce« average-fj,;^, ..opu of .pproxi»,cely 6 student, fro- ^'ch das, \t. . 

J . ^ clnss. Ilaxlisum possible score va« 

•»«ed on ranking of „e.n achievement .core at postte.t for the 21 clashes. * . 

^daee difference between total acorea « the pretest and posttc«." 

*Bascd on ranking of thk oeaa gslna for all 21 classrooms. 
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Tlble 7 

Test Iteaui exhibiting distinctive class-level patterns of jtudent performance* 



Question 
- ^ 



-.Alternatives 



1^ 1± 2 1-A 



Nature of the Response Pattern 



Pretest p-^value-.ll 
Posttest p-^alue*«,35 



As students learned about subtracting frac- 
tions » many did not learn how to obtain a 
common denomixxator before siibtracting. This 
resulted in some classes systematically 
choosing alternative 2, probably because 
they had been taught how to handle fractions, 
with a common denominator (viccurred in 5 
classes at posttest)* 3 classes exhibited 
nisteyy at posttest. 



2 + 1 

3 3 



I 3 



TT 1 



Pre p-value -.11 
Post p-^value-.26 



7^7 



28 _8 
35 14 



4 
7 



prep-value >• .07 
Post p-value».24 



The most common error for both questions is 
adding both the numerator and denominator* 
"ihis problem was not resolved at the poattest 
for several classes (9 classes on the first 
item, 13 on the second)* Both questions require 
change to mixed fractions • which students in 
many classes apparently were not taught • 

Pour classes exhibited mastery of the first " 
item and 2 classes exhibited mkstery of the 
second item at posttest 



1 
2 



2 
3 



2 
3 



_8 
12 



3 
6 



Pre p-value - .32 
Post p--value- .55 



3 
4 



2' 4 
5 9 



3 
9 



1 
6 



S'tudents in some oLasses could not expand 
fractions at the posttest (5 classes on each 
it^); They slinply added 1 to both numerator 
and denominator. 

8 classes mastered the first item at posttest 
and 2 classes exhibited mastery of the second 
item. 



Pre p-value .15 
Post p-valiie-» ,32 



What docs N equal? 
7 21 



16 



Pre i>-value » .20 
Post p-value- .33 



Lpts of class-specific confusion. Only 1 class 
exhibited mastery* Four classes systematically 
chose alternative 1, 2 classes systematically 
chose alternative 2 and 1 class systematically 
chose alternative 3 at the posttest* 

9 classes mastered the somewhat easier item ' 
3/8 - 6/N (posttest p-value -.SS) with oxay 
limited systematic ".errors. 




These are test results from selected fraction items given to 123 students In 21 
fifth grade classrooms before and after Instruction on this topic(ln most classes)- 
The data were collected as part of the Beginning Teacher Evaluation Study (Fisher 
/?;o?r* i^^^^* results reported here are based on a dissertation by Miller 
(1981; and are reported more thoroughly In Miller (1984) r- 
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'^"^1- 8 ^^H^^SION OF C^^^^^ SITJDENT READi:;; ACHIEVEMENT ON' 

TEACHER AKD MINISCHOOL LEVEL SURVEY VARIABLES^ . l> • 



Multilevel Regression 



Class Level 
Regression 




Common Minischool 
Policies 

Teacher Autonorey 
teacher Influence 
Principal Influence 



-.225 
( .487) 

-.336 
(1.445) 

1.652 
(1.601) 

.660 
(1.714) 



.360 
(1.155) 

-2.256 
(1-323) 

-1-330 
(2.292) 



-.057 
( .371) 

.563 
( .696) 




= {w'(x'x)gr^ W'X'yW , 



where 



e is the vector of regression coefficients; X is the matrix of independent 
variables; k is the number of classes; K ±1 the total number of students- 
and is a diagonal matrix of class sizes. Use of the matrix V insures that 
each classroom in the analysis will be weighted by the number oi students 
contained in it. while the overall degrees of freedom for classes will be 
preserved, 
c 

This variable was measured such that a negative coef f i^-f 
greater influence. ^ coefficient represents 



Source: Capell, p.. J. A study of alternn M,rno imoW^. ^ . , ' 

Santa Monica. cKT^^^Tc^^^^^^^^f^; JgSl! ^ ^"^^ 
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U>Umi ' tf, ftii^i I I, 



Personal 



Data Domain* (Examplas Onty} 



e 

o 

s 

o 
to 



o 
II 

w 

o 

* 

££ 
o c 

m O 

5« 



o 



• Rtasons for •nttrinc edu» 
^ c*tion proftssfon 
; • Ttaching •xptrience 
3 • ECucatJonal fu^is 


• Relatlvt amounts of time 
spent on Instruction, be- 
navior control, and routines 

• Use of behavioral objec- 
tives ^ 

• Frequency of certain 
learning activities 


ocnooi 

•'^•'•^^/•^ Importance of 
school functions (scclaf, in- 
tellectual. personal, and vo- 
cational) 

•School •^cllmaie- or v^ork 

•nvlronment 

•Major prcwema 

(•DilUy, face, sex) 


Schooling 

• Desegregation 

♦Fiscal support of public 

•ducaritVT 

•Teachers unions 

• Minimum compelen<;y 

• KOle of gfobal educatiorfr 
m the schools • 


• Dtmosraphy 

• StU-conc«p| 

• Etfucalionai asplralfcn* 

• Demography 


•Relative amounts of time 
apsnt on Inatrucllon, be« 
I;*^/?rc?"t^or, and routines 
•Difficulty of claa. content 
•Frequency of certain 
Itarnlny actlvites 
•Class -climate-. 


•Relative Importance of 
school functions 

• Evaluative ratfng 

• Major problems 

• Equality of educatJon 

5uDject-area preferences 


•Desegregation 

• Role of job experience In 
schools 

• Value of schools 


• Year* Hved In community 

• Por;i{cal beliefs 




•Relative Importance of 
school functions 
•Evaluative raHng 
•Major problems 
• Equality of education 

•^^^^^f^*"* In activities 
and decision making 

•Objtctlonable fearnino 
materials 


• Desegregation 

•Fiscal support of publk: 
s^ucatioft 

•Teachers unions 1 
•Teachers* salaries [ 

• Minimum competency 1 
•Role of global education 1 
in schools 1 




•Relative amounts of time 1 
spent on instruction, be- 
havior control, and routines 
►Use of corrective 'feed* 
t)ack 

• Use of open versus closed 
questions 

► Instnictlonaf time spent 
vith total class versus indl- 
idual versus groups " 







.^E"-"^— ^^^^^ 



Figure i 
The Schooling Tjerrain; Hap One 
SOURCE: Goodlad. Sirotnik & Overaan, 1979 
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DATA DdHAIHS 



Individual 



School 



District 



Diti 

Cttegorits i 



Personal 
(Individual) 

"c A H~ 



Data 

Sources ; 

Students 
Teachers 
Administrators 
Parents 

Students 

Teachers 

Adoinistrators 

Parents 

Classroom 

Students 

Teachers 

Administrators 

Parents 

Classrooms 

School 

Students 

Teachers 

Administrators 

Parents 

Classrooins 

Schools 

District 



Instructional 
(Classroom) 

"c A ~ 



Institutional 
(School } 

C A H 



Societal 
(Schooling) 

C A H 



Data Cateqories t 
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